{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NN复杂度\n",
    "## 空间复杂度\n",
    "层数 = 隐藏层数 + 1个输出层  \n",
    "以3-4-2为例：一个隐藏层+一个输出层=2层  \n",
    "总参数 = 总w + 总b  \n",
    "以3-4-2为例：3*4+4 + 4*2+2=26  \n",
    "## 时间复杂度\n",
    "乘加运算的次数  \n",
    "以3-4-2为例：3*4+4*2=20  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 学习率\n",
    "$$ w_{t+1} = w_t - lr*\\frac{\\partial loss}{\\partial w_t} $$\n",
    "## 指数衰减学习率\n",
    "$$ 指数衰减学习率 = 初始学习率 * 学习率衰减率^{当前轮数 / 多少轮衰减一次} $$\n",
    "eg：损失函数 loss=（w + 1）^2 | 梯度▽=（dloss）/（dw）=2w+2 | (当w=-1时，▽=0，最优)  \n",
    "init_w=5,learning_rate=0.2  \n",
    "1. | w=5 | 5-0.2*（2*5+2）=2.6  \n",
    "2. | w=2.6 | 2.6-0.2*（2*2.6+2）=1.16  \n",
    "......  \n",
    "学习率大了会震荡不收敛，学习率小了会收敛速度慢  \n",
    "这里引入指数衰减学习率这个概念：  \n",
    "指数衰减学习率是先使用较大的学习率来快速得到一个较优的解，然后随着迭代的继续,逐步减小学习率，使得模型在训练后期更加稳定  \n",
    "eg: lr = LR_BASE * LR_DECAY ** (epoch / LR_STEP)  \n",
    "非线性、可微性-优化器使用梯度下降、单调性-单层神经网络的损失函数是凸函数、近似恒等-f(x)≈x,当参数初始化为随机小值时，神经网络更稳定  \n",
    "激活函数输出为有限值时，基于梯度的优化方法更稳定  \n",
    "激活函数输出为无限值时，建议调小学习率  \n",
    "\n",
    "# 激活函数\n",
    "### Sigmoid函数\n",
    "$$ f(x) = \\frac{1}{1+e^{-x}} $$\n",
    "tf.nn.sigmoid(x)  \n",
    "#### 特点\n",
    "易造成梯度消失、输出非0均值，收敛慢、幂运算复杂度，训练时间长  \n",
    "### TanH函数\n",
    "$$ f(x) = \\frac{1-e^{-2x}}{1+e^{-2x}} $$\n",
    "tf.math.tanh(x)\n",
    "#### 特点\n",
    "输出的是0均值、易造成梯度消失、幂运算复杂度，训练时间长\n",
    "### Relu函数\n",
    "$$ f(x) = max(x,0)=\\begin{cases}0,&x<0 \\\\ x,&x>=0 \\end{cases} $$\n",
    "tf.nn.relu(x)\n",
    "#### 优点\n",
    "解决了梯度消失问题（在正区间）、只需判断输入是否大于0，计算速度快、收敛速度远快于sigmoid和tanh  \n",
    "#### 缺点\n",
    "输出非0均值，收敛慢、Dead RelU问题：某些神经元可能永远不被激活，导致相应的参数永远不能被更新（输入负数特征过多，减少学习率，减少参数分布的巨大变化）。\n",
    "### Leaky Relu函数\n",
    "$$ f(x) = max(αx,x)=\\begin{cases}αx,&x<0 \\\\ x,&x>=0 \\end{cases} $$\n",
    "tf.nn.leaky_relu(x)\n",
    "#### 特点\n",
    "理论上leaky_relu有relu的优点，但不会出现dead relu问题，实际上效果不一定优于relu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tips\n",
    "首选relu激活函数  \n",
    "学习率设置较小  \n",
    "输入特征标准化，即让输入特征满足以0为均值，1为标准差的正态分布  \n",
    "初始参数中心化，即让随机生成的参数满足以0为均值，$$ \\sqrt{\\frac{2}{当前层输入特征个数}} $$为标准差的正态分布"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 损失函数（loss）：预测值（y）与已知答案（y_）的差距\n",
    "NN优化的目标：loss最小\n",
    "## 均方误差MSE（Mean Squared Error）\n",
    "$$ mse(y_,y)=\\frac{Σ_{i=1}^n(y-y\\_)^2}{n} $$\n",
    "### loss_mse=tf.reduce_mean(tf.square(y_-y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "After 0 training steps,w1 is\n",
      "[[ 0.36442968]\n",
      " [-0.9614987 ]] \n",
      "\n",
      "After 500 training steps,w1 is\n",
      "[[1.0837942 ]\n",
      " [0.07894276]] \n",
      "\n",
      "After 1000 training steps,w1 is\n",
      "[[1.272613  ]\n",
      " [0.47505528]] \n",
      "\n",
      "After 1500 training steps,w1 is\n",
      "[[1.2926469 ]\n",
      " [0.64627784]] \n",
      "\n",
      "After 2000 training steps,w1 is\n",
      "[[1.2637599 ]\n",
      " [0.73507214]] \n",
      "\n",
      "After 2500 training steps,w1 is\n",
      "[[1.2248375]\n",
      " [0.7905472]] \n",
      "\n",
      "After 3000 training steps,w1 is\n",
      "[[1.1877908]\n",
      " [0.8302174]] \n",
      "\n",
      "After 3500 training steps,w1 is\n",
      "[[1.1556823]\n",
      " [0.8607971]] \n",
      "\n",
      "After 4000 training steps,w1 is\n",
      "[[1.1287668]\n",
      " [0.8852214]] \n",
      "\n",
      "After 4500 training steps,w1 is\n",
      "[[1.1064948]\n",
      " [0.9050342]] \n",
      "\n",
      "After 5000 training steps,w1 is\n",
      "[[1.0881608 ]\n",
      " [0.92121136]] \n",
      "\n",
      "After 5500 training steps,w1 is\n",
      "[[1.0730999]\n",
      " [0.9344564]] \n",
      "\n",
      "After 6000 training steps,w1 is\n",
      "[[1.0607392]\n",
      " [0.9453121]] \n",
      "\n",
      "After 6500 training steps,w1 is\n",
      "[[1.0505978]\n",
      " [0.9542137]] \n",
      "\n",
      "After 7000 training steps,w1 is\n",
      "[[1.0422784]\n",
      " [0.9615144]] \n",
      "\n",
      "After 7500 training steps,w1 is\n",
      "[[1.0354537 ]\n",
      " [0.96750236]] \n",
      "\n",
      "After 8000 training steps,w1 is\n",
      "[[1.0298562 ]\n",
      " [0.97241384]] \n",
      "\n",
      "After 8500 training steps,w1 is\n",
      "[[1.0252641 ]\n",
      " [0.97644275]] \n",
      "\n",
      "After 9000 training steps,w1 is\n",
      "[[1.021498 ]\n",
      " [0.9797474]] \n",
      "\n",
      "After 9500 training steps,w1 is\n",
      "[[1.0184087]\n",
      " [0.982458 ]] \n",
      "\n",
      "After 10000 training steps,w1 is\n",
      "[[1.0158739 ]\n",
      " [0.98468167]] \n",
      "\n",
      "After 10500 training steps,w1 is\n",
      "[[1.013795 ]\n",
      " [0.9865052]] \n",
      "\n",
      "After 11000 training steps,w1 is\n",
      "[[1.0120912]\n",
      " [0.9880011]] \n",
      "\n",
      "After 11500 training steps,w1 is\n",
      "[[1.0106928]\n",
      " [0.9892279]] \n",
      "\n",
      "After 12000 training steps,w1 is\n",
      "[[1.0095451 ]\n",
      " [0.99023455]] \n",
      "\n",
      "After 12500 training steps,w1 is\n",
      "[[1.0086052 ]\n",
      " [0.99105936]] \n",
      "\n",
      "After 13000 training steps,w1 is\n",
      "[[1.0078337 ]\n",
      " [0.99173695]] \n",
      "\n",
      "After 13500 training steps,w1 is\n",
      "[[1.0071998 ]\n",
      " [0.99229133]] \n",
      "\n",
      "After 14000 training steps,w1 is\n",
      "[[1.0066837 ]\n",
      " [0.99274653]] \n",
      "\n",
      "After 14500 training steps,w1 is\n",
      "[[1.0062544]\n",
      " [0.9931205]] \n",
      "\n",
      "Final w1 is: [[1.0059096 ]\n",
      " [0.99342567]]\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "seed = 23455     #随机种子\n",
    "rdm = np.random.RandomState(seed)\n",
    "x = rdm.rand(32,2)\n",
    "y_ = [[x1+x2+(rdm.rand()/10.0-0.05)] for (x1,x2) in x]\n",
    "\n",
    "#x转变数据类型\n",
    "x = tf.cast(x,dtype=tf.float32)\n",
    "w1 = tf.Variable(tf.random.normal([2,1],stddev=1,seed=1))\n",
    "\n",
    "epoch = 15000#循环迭代次数\n",
    "lr = 0.002#学习率\n",
    "\n",
    "for epoch in range(epoch):\n",
    "    with tf.GradientTape() as tape:#梯度下降\n",
    "        y = tf.matmul(x,w1)#前向传播计算结果\n",
    "        loss_mse = tf.reduce_mean(tf.square(y_-y))#损失函数：均方误差\n",
    "        \n",
    "    grads = tape.gradient(loss_mse,w1)#损失函数对待训练参数w1求偏导\n",
    "    w1.assign_sub(lr*grads)#更新参数w1\n",
    "    \n",
    "    if epoch %500 == 0:\n",
    "        print(\"After %d training steps,w1 is\"%(epoch))\n",
    "        print(w1.numpy(),\"\\n\")\n",
    "        \n",
    "print(\"Final w1 is:\",w1.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 自定义损失函数\n",
    "自定义损失函数  $$ loss(y_,y) = Σ_nf(y_,y) $$\n",
    "### 自定义成本与利润\n",
    "$$ f(y_,y) = \\begin{cases}PROFIT*(y\\_-y),&y<y\\_ \\\\ COST*(y-y\\_),&y>y\\_ \\end{cases} $$\n",
    "### loss_zdy = tf.reduce_sum(tf.where(tf.greater(y,y_),COST*(y-y_),PROFIT*(y_-y)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "After 0 training steps,w1 is\n",
      "[[29.671358]\n",
      " [32.047916]] \n",
      "\n",
      "After 500 training steps,w1 is\n",
      "[[14.436411]\n",
      " [15.782992]] \n",
      "\n",
      "After 1000 training steps,w1 is\n",
      "[[1.1549116]\n",
      " [2.1614442]] \n",
      "\n",
      "After 1500 training steps,w1 is\n",
      "[[1.3539518]\n",
      " [1.2449582]] \n",
      "\n",
      "After 2000 training steps,w1 is\n",
      "[[1.5100361]\n",
      " [1.0314224]] \n",
      "\n",
      "After 2500 training steps,w1 is\n",
      "[[1.2925394]\n",
      " [2.2895539]] \n",
      "\n",
      "After 3000 training steps,w1 is\n",
      "[[1.297658]\n",
      " [1.275389]] \n",
      "\n",
      "After 3500 training steps,w1 is\n",
      "[[1.6664945]\n",
      " [2.6464522]] \n",
      "\n",
      "After 4000 training steps,w1 is\n",
      "[[1.2408148]\n",
      " [2.1398804]] \n",
      "\n",
      "After 4500 training steps,w1 is\n",
      "[[1.2029781]\n",
      " [1.828666 ]] \n",
      "\n",
      "After 5000 training steps,w1 is\n",
      "[[1.363632 ]\n",
      " [1.4350257]] \n",
      "\n",
      "After 5500 training steps,w1 is\n",
      "[[1.5197163]\n",
      " [1.2214899]] \n",
      "\n",
      "After 6000 training steps,w1 is\n",
      "[[1.6900622]\n",
      " [2.6749787]] \n",
      "\n",
      "After 6500 training steps,w1 is\n",
      "[[1.0750307]\n",
      " [1.8906238]] \n",
      "\n",
      "After 7000 training steps,w1 is\n",
      "[[1.2311151]\n",
      " [1.677088 ]] \n",
      "\n",
      "After 7500 training steps,w1 is\n",
      "[[1.3871995]\n",
      " [1.4635522]] \n",
      "\n",
      "After 8000 training steps,w1 is\n",
      "[[1.5478532]\n",
      " [1.069912 ]] \n",
      "\n",
      "After 8500 training steps,w1 is\n",
      "[[1.3919889]\n",
      " [4.117639 ]] \n",
      "\n",
      "After 9000 training steps,w1 is\n",
      "[[1.3354752]\n",
      " [1.3138793]] \n",
      "\n",
      "After 9500 training steps,w1 is\n",
      "[[1.165503]\n",
      " [1.688956]] \n",
      "\n",
      "Final w1 is: [[1.3520573]\n",
      " [1.5079508]]\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "seed = 23455     #随机种子\n",
    "COST = 1\n",
    "PROFIT = 999\n",
    "\n",
    "rdm = np.random.RandomState(seed)\n",
    "x = rdm.rand(32,2)\n",
    "y_ = [[x1+x2+(rdm.rand()/10.0-0.05)] for (x1,x2) in x]\n",
    "\n",
    "#x转变数据类型\n",
    "x = tf.cast(x,dtype=tf.float32)\n",
    "w1 = tf.Variable(tf.random.normal([2,1],stddev=1,seed=1))\n",
    "\n",
    "epoch = 10000#循环迭代次数\n",
    "lr = 0.002#学习率\n",
    "\n",
    "for epoch in range(epoch):\n",
    "    with tf.GradientTape() as tape:#梯度下降\n",
    "        y = tf.matmul(x,w1)#前向传播计算结果\n",
    "        loss_zdy = tf.reduce_sum(tf.where(tf.greater(y,y_),(y-y_)*COST,(y_-y)*PROFIT))\n",
    "        \n",
    "    grads = tape.gradient(loss_zdy,w1)#损失函数对待训练参数w1求偏导\n",
    "    w1.assign_sub(lr*grads)#更新参数w1\n",
    "    \n",
    "    if epoch %500 == 0:\n",
    "        print(\"After %d training steps,w1 is\"%(epoch))\n",
    "        print(w1.numpy(),\"\\n\")\n",
    "        \n",
    "print(\"Final w1 is:\",w1.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 交叉熵损失函数CE(Cross Entropy)：表征两个概率分布之间的距离\n",
    "$$ H(y_,y) = -Σy\\_*ln y $$\n",
    "### tf.losses.categorical_crossentropy(y_,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss_ce1: tf.Tensor(0.5108256, shape=(), dtype=float32)\n",
      "loss_ce2: tf.Tensor(0.22314353, shape=(), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "loss_ce1 = tf.losses.categorical_crossentropy([1,0],[0.6,0.4])\n",
    "loss_ce2 = tf.losses.categorical_crossentropy([1,0],[0.8,0.2])\n",
    "print(\"loss_ce1:\",loss_ce1)\n",
    "print(\"loss_ce2:\",loss_ce2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## softmax与交叉熵结合\n",
    "输出先过softmax函数，再计算y与y_的交叉熵损失函数\n",
    "### tf.nn.softmax_cross_entropy_with_logits(y_,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "ename": "NotFoundError",
     "evalue": "Could not find valid device for node.\nNode:{{node Softmax}}\nAll kernels registered for op Softmax :\n  device='CPU'; T in [DT_HALF]\n  device='CPU'; T in [DT_FLOAT]\n  device='CPU'; T in [DT_DOUBLE]\n [Op:Softmax]",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNotFoundError\u001b[0m                             Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-36-148c674bdbbe>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[0my\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0marray\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m10\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 4\u001b[1;33m \u001b[0my_pro\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mnn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msoftmax\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0my\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      5\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      6\u001b[0m \u001b[0mloss_ce1\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mlosses\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcategorical_crossentropy\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0my_\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0my_pro\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mC:\\development\\Anaconda\\envs\\TF2.1\\lib\\site-packages\\tensorflow_core\\python\\ops\\nn_ops.py\u001b[0m in \u001b[0;36msoftmax_v2\u001b[1;34m(logits, axis, name)\u001b[0m\n\u001b[0;32m   3036\u001b[0m   \u001b[1;32mif\u001b[0m \u001b[0maxis\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   3037\u001b[0m     \u001b[0maxis\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m-\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 3038\u001b[1;33m   \u001b[1;32mreturn\u001b[0m \u001b[0m_softmax\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlogits\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mgen_nn_ops\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msoftmax\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mname\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   3039\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   3040\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mC:\\development\\Anaconda\\envs\\TF2.1\\lib\\site-packages\\tensorflow_core\\python\\ops\\nn_ops.py\u001b[0m in \u001b[0;36m_softmax\u001b[1;34m(logits, compute_op, dim, name)\u001b[0m\n\u001b[0;32m   2929\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   2930\u001b[0m   \u001b[1;32mif\u001b[0m \u001b[0mis_last_dim\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 2931\u001b[1;33m     \u001b[1;32mreturn\u001b[0m \u001b[0mcompute_op\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlogits\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mname\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mname\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   2932\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   2933\u001b[0m   \u001b[0mdim_val\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdim\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mC:\\development\\Anaconda\\envs\\TF2.1\\lib\\site-packages\\tensorflow_core\\python\\ops\\gen_nn_ops.py\u001b[0m in \u001b[0;36msoftmax\u001b[1;34m(logits, name)\u001b[0m\n\u001b[0;32m  10417\u001b[0m         \u001b[1;32mpass\u001b[0m  \u001b[1;31m# Add nodes to the TensorFlow graph.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m  10418\u001b[0m     \u001b[1;32mexcept\u001b[0m \u001b[0m_core\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_NotOkStatusException\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m> 10419\u001b[1;33m       \u001b[0m_ops\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mraise_from_not_ok_status\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mname\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m  10420\u001b[0m   \u001b[1;31m# Add nodes to the TensorFlow graph.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m  10421\u001b[0m   _, _, _op, _outputs = _op_def_library._apply_op_helper(\n",
      "\u001b[1;32mC:\\development\\Anaconda\\envs\\TF2.1\\lib\\site-packages\\tensorflow_core\\python\\framework\\ops.py\u001b[0m in \u001b[0;36mraise_from_not_ok_status\u001b[1;34m(e, name)\u001b[0m\n\u001b[0;32m   6604\u001b[0m   \u001b[0mmessage\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmessage\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;33m(\u001b[0m\u001b[1;34m\" name: \"\u001b[0m \u001b[1;33m+\u001b[0m \u001b[0mname\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0mname\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[1;32mNone\u001b[0m \u001b[1;32melse\u001b[0m \u001b[1;34m\"\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   6605\u001b[0m   \u001b[1;31m# pylint: disable=protected-access\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 6606\u001b[1;33m   \u001b[0msix\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mraise_from\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mcore\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_status_to_exception\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcode\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmessage\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   6607\u001b[0m   \u001b[1;31m# pylint: enable=protected-access\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   6608\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mC:\\development\\Anaconda\\envs\\TF2.1\\lib\\site-packages\\six.py\u001b[0m in \u001b[0;36mraise_from\u001b[1;34m(value, from_value)\u001b[0m\n",
      "\u001b[1;31mNotFoundError\u001b[0m: Could not find valid device for node.\nNode:{{node Softmax}}\nAll kernels registered for op Softmax :\n  device='CPU'; T in [DT_HALF]\n  device='CPU'; T in [DT_FLOAT]\n  device='CPU'; T in [DT_DOUBLE]\n [Op:Softmax]"
     ]
    }
   ],
   "source": [
    "y_ = np.array([[1,0,0],[0,1,0],[0,0,1],[1,0,0],[0,1,0]])\n",
    "y = np.array([[1,0,0],[0,1,0],[0,10,1],[1,0,0],[0,1,0]])\n",
    "\n",
    "y_pro = tf.nn.softmax(y)\n",
    "\n",
    "loss_ce1 = tf.losses.categorical_crossentropy(y_,y_pro)\n",
    "loss_ce2 = tf.nn.softmax_cross_entropy_with_logits(y_,y)\n",
    "print(\"分步计算：\",loss_ce1)\n",
    "print(\"合步计算：\",loss_ce2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
